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The nature of the insulating ground state in quasi-two-dimensional organic conductor A- 
(BETS)2GaX 2 Y4_ 2 , where the existence of a spin gap is suggested by susceptibility measure- 
ment, has been studied theoretically. Hartree-Fock calculations at absolute zero temperature 
show that if the on-site Coulomb interaction exceeds some critical value, then antiferromag- 
netic spin ordering emerges and eventually leads to an insulating state which can be considered 
as a two-dimensional localized spin system. Based on the quantum Monte Carlo simulations 
of Katoh and Imada, we will show that this system locates near the boundary between the 
antiferromagnetic phase and the spin gap phase, so that experimental facts can be explained. 

KEYWORDS: BETS, spin gap, Hartree-Fock approximation, on-site Coulomb interaction, antiferromagnetic 
spin ordering 

Organic conductors made of BEDT-TSeF (bisethylenedithiotetraselenafulvalene, abbreviated as 
BETS), which is a seleno-analog of the well-known BEDT-TTF (abbreviated as ET), have two- 
dimensional (2D) character as ET compounds does. Among them, a new series of superconductors 
A-(BETS)2GaX z Y4_ z (X,Y=F,Cl,Br) exhibit interesting electronic properties. By recent resistivity 
and magnetic susceptibility measurements with different z- values and different pressures, the phase 
diagram of A-(BETS)2GaX z Y4_ z was proposed^) as shown in Fig. 1. The substitution of X and Y 
to smaller atoms, the decrease of the z- value when X is a larger atom than Y or condition of applied 
pressure corresponds to the reduction of the unit cell volume V . Since the magnetic susceptibility^ 
showed a steep decrease without anisotropy by lowering the temperature in the compounds that 
exhibit an insulating ground state, this phase, which is next to the superconducting phase, is 
considered to be a nonmagnetic insulating (NMI) phase. We can see that the general feature of 
this phase diagram is analogous to the proposed phase diagram of ft-(ET)2X,i) where the 10-K 
superconducting phase is situated near the Mott antiferromagnetic insulating (AFI) phase,& if the 
NMI phase in the present case is replaced by the AFI phase. 

* E-mail: hseo@watson.phys.s.u-tokyo. ac.jp 
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Fig. 1. Experimental phase diagram of A-(BETS)2GaX 2 Y4-z on the plane of unit cell volume (V) and temperature 
(T).Q) NMI, SC, PI and PM stands for the nonmagnetic insulating, superconducting, paramagnetic insulating and 
paramagnetic metallic phases, respectively. 

A-(BETS)2GaX^Y4_ z consists of four BETS molecules in a unit cell and take layered structure 
of 2D BETS network, whose structure in the donor plane is schematically shown in Fig. 2. It is 
seen in Fig. 2 that there exist dimers (1-2) and (3-4) which are symmetrically equivalent since 
their intradimer transfer integral t& is twice as large as the others. We note that values of and 
tc differ due to the crystal structure. BETS is electrically expressed as BETS +1 / 2 in this family, 
and then there are two holes in a unit cell. The extended Hiickel band calculations^ predict that 
these A-type compounds are metallic with a 2D Fermi surface very similar to those of the ft-type 
conductors, despite their large differences in the molecular arrangement. The partially- filled higher 
two bands and the fully-occupied lower two bands are separated so we can consider the system 
effectively half-filled, thus strong correlation can lead the system to a Mott insulator. Since the 
behavior in the magnetic susceptibility in the insulating state is reminiscent of that in the quasi-one- 
dimensional organic spin-Peierls compounds such as (TMTTF^PFgi) which also has an effectively 
half-filled band, we may speculate that the NMI phase is due to the spin-Peierls transition which 
is characteristic of one-dimensional materials, yet the 2D character of these BETS compounds may 
contradict this conjecture. Our aim in this paper is to clarify the origin of this NMI phase. 

In order to study the electronic structure of these BETS salts, we only consider their 2D donor 
plane (Fig. 2) which consists of transfer integrals between BETS molecules and the on-site Coulomb 
interaction U on BETS molecule. The Hamiltonian is written as 



2 



Fig. 2. A schematic view of the structure of the donor plane in A-(BETS)2GaX z Y4_ z . The values of transfer 
integrals for Y=C1, z — are given by t — Es, where s is the value of the overlap integral from ref. 1 and 
E = -lOeV. 



H= {hMa^cr + h.c.) + U^n^n ib (1) 

<i,j> & % 

where < z, j > denotes the neighbor site pair, a is a spin index which takes j and j, n{ G and a\ a 
(dicr) denote the number operator and the creation (annihilation) operator for the electron of spin 
a at the zth site, respectively. 

We treat U in the Hartree-Fock (HF) approximation, as in ref. 3, and assume that the periodicity 
of the electron system along the a-direction is the same as that of the original lattice. As for the 
c-direction, we considered two cases; one with the same periodicity as that of the original lattice 
system, i.e. the paramagnetic or ferromagnetic solutions along the c-direction, or one two-fold that 
of the original lattice system, i.e. the antiferromagnetic solution along the c-direction. Thus, there 
are four or eight different sites in a unit cell and the HF Hamiltonian in fc-space is given by 
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where m = 4 or m = 8 according to the unit cell size, a is opposite to a and a v ^ a is the Fourier 
transform of a^, i.e. a v ^ a — 



^ N — Y2 a e 1 ' a (ais)a an d N ce \\ is the total number of unit cells, a 
and v denote the cell number and the number of the molecule, respectively. The ho term is a 
matrix which comes from the first term of eq. (1) and is diagonalized to lead to almost the same 
dispersionas that given by the extended Hiickel band calculation.0) 
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In eq. (2), the electron densities for each spin, n VG {y = 1, 2, • • • , m and a =|, j), are determined 
self-consistently by 



where N e is the total number of electrons in the unit cell in the highest occupied molecular orbital 
(HOMO) levels. (N e = 6 for m=4 or N e = 12 for m=8 in the system currently considered.) 

The calculations indicate that an antiferromagnetic (AF) order appears when U exceeds some 
critical value and it affects the degree of the band overlap. The calculated absolute magnitude of 
spin moment, S Z1 for molecules 1 and 2 and the band gap as a function of U are shown in Fig. 3. We 
use the term 'band gap' as a value of the lowest amount of energy for the highest (second) partially 
or fully filled band minus the highest amount of energy for the second (third) partially or fully filled 
band for m=4 (m=8), so a negative band gap implies a finite band overlap. The results indicate 
that there exist two phase transitions; a 2nd-order transition between a paramagnetic metal state 
and an AF metal (AFM) state at U = U c c± 0.279 eV and another 2nd-order transition from AFM 
state to AFI state at U = Umi — 0.296 eV. The hole density at each site is slightly different from 
+0.5 but is similar. In the AFM and AFI phases, the configuration of spin alignments is shown in 
the inset of Fig. 3. The direction of the spin moments inside the dimers are the same and a 2D 
AF ordering occurs between dimers. We note that S z (l) — —5^(4), S z (2) = —S z (3) in each row 
along the a-direction, and that S z (l) and S z (2) differ by a few % mainly due to the differences of 
the transfer integrals ts and tc- 

One can extract the essence by taking each dimer as a unit together with the effective overlap 
integrals between dimers. This view is called the dimer model in the case of ^-(ET^X.!'!) The above 
results show that in the large U region, where the BETS compounds are considered to be located, 
the system can be viewed as a 2D localized spin system, therefore, we can construct an effective 
2D Heisenberg spin model as schematically shown in Fig. 4. We estimate the superexchange 
coupling constants, J, using the relation J ~ At 2 /V dimer (Note that Udimer is the effective Coulomb 
interaction between two holes in a dimer), and conclude that the magnitudes of the interactions 
between the spins are Jc — 0.52J#, Jj_ ~ 0.72 J#, J s ~ 0.19 Jb and J t ~ 0.02J#. If we neglect J s 
and J tl which are noticeably smaller than the others, the system can be viewed as dimerized AF 
Heisenberg chains along the a-direction coupled by interchain coupling J± by which the Hamiltonian 
is described as 




(3) 



They are calculated together with the equation for the Fermi energy implicitly given by 
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Fig. 3. U dependence of the absolute magnitude of spin moments, S z , and the band gap (negative value implies a 
band overlap). 

where Si is the 5=1/2 spin operator on the ith site and a and c denote the unit vector in the 
a-direction and the c-direction, respectively. 

In 2D quantum spin systems, quantum fluctuations, which we have not taken into account until 
now, result in the competition between the antiferromagnetic ordered state and the disordered spin 
liquid state which has a spin gap. In our model there are two key parameters which determine 
which state will be realized; the degree of dimerization, i.e. the difference between Jc and J#, 
and the interchain coupling J±. As a general tendency, the spin gap phase is stabilized in the 
region where the degree of dimerization is large enough. The interchain coupling J± prefers the AF 
state when J±< Jc, Jb- In our system, however, J± ~ J c , J B and it is not obvious which state is 
stabilized. Katoh and ImadaEI carried out quantum Monte Carlo simulations to the Hamiltonian 
(5), and determined the phase diagram in a parameter space of the magnitude of interchain coupling 
and the degree of dimerization (in their notation Jb — J(l + S) and Jc — J (I — S), S: degree of 
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Fig. 4. Effective 2D Heisenberg model for localized spins in the dimer model for A-(BETS)2GaBr x Cl4-; 



dimerization). According to their results, our spin system is located in the region close to the 
boundary between the AF phase and the spin gap phase, but slightly in the region of the AF state. 
It is expected that J s , which has been neglected so far, pushes the system toward the spin gap phase. 
Considering the magnetic susceptibility experiments]^ the NMI phase of A-(BETS)2GaX z Y4_ z is 
then thought to be in the spin gap regime. 

We can either see the system as 2-leg ladders along the c-direction whose interaction along 
the rungs is Jb and that along the legs is J±, coupled by interladder coupling Jq. Recently, a 
similar system, where two-leg ladders are coupled three-dimensionally, was studied.!' B For example, 
in LaCuO2.5,!'!'0) the static susceptibility!! suggests a spin liquid state with a spin gap, while 
NMR& and //SR0) measurements indicate transition to a magnetically ordered phase. Theoritical 
studies!' % indicate that in a system located in the AF phase near the transition point to the 
spin liquid state, the static susceptibility has the form x(T) = xo + &T 2 : , with a small xo which 
will not be identified easily by experiment. Therefore, we cannot exclude the possibility that A- 
(BETS)2GaX z Y4_ z is such an unconventional AF, close to the quantum critical point. For the 
moment we cannot decide beween the two possibilities, the 2D spin-gap phase or the AF close 
to criticality, and must wait for further experiments, such as NMR and /xSR measurements for 
clarification. 

Increasing Jq corresponds to the decrease of the degree of dimerization in the picture of dimerized 
Heisenberg chains, and to the increase of the interladder coupling in the picture of ladders system. 
In either picture, we see that, if Jq increases, the system moves in the direction of the AF ordered 
state. Therefore, if one can synthesize compounds analogous to A^BETS^GaX^Y^^, but with a 
smaller degree of dimerization, the system should have AF ground state. By controlling the systems 
in such a way, we will be able to study quantum spin systems systematically. 

In summary, we have studied the nature of the nonmagnetic insulating phase in A- 
(BETS)2GaX z Y4_ z . Hartree-Fock calculations suggest that the on-site Coulomb interaction U 
causes an AF insulating state, which is a Mott insulator with one hole per dimer. Deducing from 
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these results, we constructed an effective 2D localized dimer spin model with different exchange 
couplings. It is argued that quantum effects make this system a disordered spin liquid state with 
a spin gap, and we infer that the system is located close to the quantum critical point. 
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tp -0.021 eV 

tq -0.040 eV 

t r 0.034 eV 
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